Skip to content

[fix](local shuffle) fix bucket shuffle exchange incorrectly marked as serial in pooling mode#62054

Open
924060929 wants to merge 1 commit intoapache:masterfrom
924060929:opt_bucket_shuffle_exchange
Open

[fix](local shuffle) fix bucket shuffle exchange incorrectly marked as serial in pooling mode#62054
924060929 wants to merge 1 commit intoapache:masterfrom
924060929:opt_bucket_shuffle_exchange

Conversation

@924060929
Copy link
Copy Markdown
Contributor

@924060929 924060929 commented Apr 2, 2026 •
edited
Loading

  1. ExchangeNode.isSerialOperator(): move useSerialSource() guard into isSerialOperator()
       so that is_serial is a self-contained per-operator property. Only UNPARTITIONED or
       use_serial_exchange exchanges in pooling mode are serial. Bucket shuffle / hash
       exchanges are never serial — scan seriality is handled by the scan pipeline independently.

  2. DistributePlanner.filterInstancesWhichCanReceiveDataFromRemote(): use
       linkNode.isSerialOperator() to decide whether to dedupe to first-per-worker.
       Remove LocalShuffleAssignedJob type check and enableShareHashTableForBroadcastJoin
       parameter — serial exchange semantics already capture the correct behavior.

  3. Add unit tests for both changes.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Copy Markdown
Contributor

Thearas commented Apr 2, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@924060929 924060929 force-pushed the opt_bucket_shuffle_exchange branch from e935b63 to 4c3e765 Compare April 2, 2026 14:11
@924060929
Copy link
Copy Markdown
Contributor Author

924060929 commented Apr 2, 2026

run buildall

@924060929
[fix](local shuffle) fix bucket shuffle exchange incorrectly marked a…
122110b 
…s serial in pooling mode

1. ExchangeNode.toThrift: is_serial = isSerialOperator() && useSerialSource().
   isSerialOperator() is pure operator-level: only UNPARTITIONED or use_serial_exchange.
   useSerialSource() is the fragment-level pooling guard, kept in toThrift (not in
   isSerialOperator) to avoid infinite recursion: useSerialSource() calls
   planRoot.isSerialOperator() internally.

2. DistributePlanner.filterInstancesWhichCanReceiveDataFromRemote(): use
   linkNode.isSerialOperator() && linkNode.getFragment().useSerialSource() to decide
   whether to dedupe to first-per-worker destinations. Bucket shuffle / hash exchanges
   are never serial — scan seriality is handled by the scan pipeline independently.

3. Add unit tests for both changes.
@924060929 924060929 force-pushed the opt_bucket_shuffle_exchange branch from 4c3e765 to 122110b Compare April 2, 2026 16:09
@924060929
Copy link
Copy Markdown
Contributor Author

924060929 commented Apr 2, 2026

run buildall

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented Apr 2, 2026

FE UT Coverage Report

Increment line coverage 66.67% (6/9) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link
Copy Markdown

doris-robot commented Apr 2, 2026

TPC-H: Total hot run time: 16520 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 122110b98440a485a3f5683e5e0ca75e8eb35d1d, data reload: false

------ Round 1 ----------------------------------
customer	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:43	NULL	utf-8	NULL	NULL	
lineitem	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:38:59	NULL	utf-8	NULL	NULL	
supplier	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:25	NULL	utf-8	NULL	NULL	
region	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:24	NULL	utf-8	NULL	NULL	
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4565	3646	3602	3602
q2	q3	3533	656	663	656
q4	843	340	355	340
q5	2939	1295	1124	1124
q6	289	150	123	123
q7	q8	7862	1261	1161	1161
q9	q10	4624	1758	1730	1730
q11	370	266	254	254
q12	615	676	499	499
q13	2451	2758	2138	2138
q14	276	285	254	254
q15	q16	837	816	766	766
q17	946	996	872	872
q18	q19	5910	960	721	721
q20	q21	3102	1967	2050	1967
q22	436	357	313	313
Total cold run time: 39598 ms
Total hot run time: 16520 ms

----- Round 2, with runtime_filter_mode=off -----
customer	Doris	NULL	NULL	15000000	92	1381653732	NULL	4374759	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:43	NULL	utf-8	NULL	NULL	
lineitem	Doris	NULL	NULL	600037902	33	19843441616	NULL	61784740	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:38:59	NULL	utf-8	NULL	NULL	
supplier	Doris	NULL	NULL	1000000	87	87519212	NULL	194931	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:25	NULL	utf-8	NULL	NULL	
region	Doris	NULL	NULL	5	240	1201	NULL	147	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:24	NULL	utf-8	NULL	NULL	
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	3896	3883	3878	3878
q2	q3	5073	4329	4305	4305
q4	1936	2047	1325	1325
q5	4921	5014	5047	5014
q6	178	154	119	119
q7	q8	8733	2919	2931	2919
q9	q10	4903	4131	4189	4131
q11	529	372	368	368
q12	647	684	477	477
q13	2417	2825	2123	2123
q14	273	275	249	249
q15	q16	730	741	672	672
q17	1188	1153	1150	1150
q18	q19	4842	885	856	856
q20	q21	5346	5079	4569	4569
q22	513	463	401	401
Total cold run time: 46125 ms
Total hot run time: 32556 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Apr 3, 2026

TPC-DS: Total hot run time: 73983 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 122110b98440a485a3f5683e5e0ca75e8eb35d1d, data reload: false

query5	4341	638	506	506
query6	348	231	218	218
query7	4244	554	336	336
query8	337	238	226	226
query9	8723	3897	3901	3897
query10	524	405	362	362
query11	6808	5522	5127	5127
query12	
query13	1328	430	418	418
query14	5823	4812	4726	4726
query14_1	
query15	354	199	192	192
query16	
query17	
query18	
query19	287	200	194	194
query20	
query21	279	118	116	116
query22	
query23	
query23_1	
query24	
query24_1	
query25	
query26	868	187	178	178
query27	3130	380	371	371
query28	3040	1891	1916	1891
query29	
query30	349	209	197	197
query31	
query32	357	76	78	76
query33	544	307	304	304
query34	2958	677	648	648
query35	792	687	649	649
query36	1246	1190	1092	1092
query37	350	102	88	88
query38	
query39	972	845	855	845
query39_1	838	830	837	830
query40	291	138	146	138
query41	76	63	63	63
query42	308	285	267	267
query43	375	271	280	271
query44	
query45	211	193	195	193
query46	
query47	
query48	753	319	313	313
query49	668	460	453	453
query50	1156	246	221	221
query51	4361	4395	4193	4193
query52	279	285	271	271
query53	333	356	272	272
query54	420	298	309	298
query55	102	97	98	97
query56	338	338	305	305
query57	
query58	366	279	282	279
query59	3292	2716	2728	2716
query60	348	328	311	311
query61	157	151	153	151
query62	707	575	550	550
query63	306	267	273	267
query64	6664	1320	1080	1080
query65	
query66	2198	488	384	384
query67	
query68	
query69	419	317	298	298
query70	1023	1004	999	999
query71	359	348	315	315
query72	
query73	1217	459	440	440
query74	
query75	3814	2986	2985	2985
query76	2659	994	863	863
query77	396	403	340	340
query78	11344	11202	10690	10690
query79	1522	1166	856	856
query80	824	746	669	669
query81	456	287	240	240
query82	1320	156	121	121
query83	368	296	274	274
query84	314	153	120	120
query85	988	514	466	466
query86	396	341	322	322
query87	
query88	3487	2719	2692	2692
query89	507	368	370	368
query90	1886	176	167	167
query91	184	143	161	143
query92	77	73	71	71
query93	897	825	503	503
query94	
query95	756	432	366	366
query96	1110	373	311	311
query97	2700	2540	2521	2521
query98	
query99	1245	982	962	962
Total cold run time: 146059 ms
Total hot run time: 73983 ms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@924060929 @Thearas @hello-stephen @doris-robot